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This study analyzed data on students in three Ohio school districts who had completed 
grades 8 and 9 to determine which data elements were the most accurate indicators 
of students’ failure to graduate from high school on time. The most accurate indicators 
varied by district and grade level, which underscores the importance of having school 
systems verify the accuracy of early warning indicators in their local contexts. School 
systems consider multiple factors when designing early warning systems, including the 
data available on students at different grade levels, the optimal cutpoints for the data 
elements, the consistency and accuracy of the indicators, and the possibility of using 
combinations of indicators for predicting failure to graduate on time. In short, the best 
indicators of failure to graduate on time for one district and grade level may not be the 
best for other districts and grade levels. 


This brief summarizes the findings of Stuit, D., O’Cummings, M., Norbury, H., Heppen, J., Dhillon, 
S., Lindsay, J., & Zhu, B. (2016). Identifying early warning indicators in three Ohio school districts (REL 
2016-118). Washington, DC: U.S. Department of Education, Institute of Education Sciences, Nation- 
al Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory 
Midwest. That report is available at http://ies.ed.gov/ncee/edlabs/projects/project.asp?ProjectID=358. 
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Why this study? 


Information from early warning systems can help educators target resources and interventions to students 
at the greatest risk of not graduating or not graduating on time (Heppen & Therriault, 2008; Jerald, 2006; 
Kennelly & Monrad, 2007; Neild, Balfanz, & Herzog, 2007; Pinkus, 2008). But how can districts determine 
which types of student data to use for their early warning systems? This study, conducted by Regional 
Educational Laboratory Midwest in partnership with the Midwest Dropout Prevention Research Alliance, 
identified valid grade 8 and 9 early warning indicators developed from datasets collected from three school 
districts that vary in size, urbanicity, and the characteristics of their student populations. 


Analysis of data from large urban districts has enabled researchers to identify indicators that predict whether 
middle school and grade 9 students will graduate on time. For middle school students, attendance, course 
grades, and behavior such as out-of-school suspensions have been frequently identified as early indicators of 
high school graduation outcomes (Balfanz, 2009; Balfanz & Herzog, 2005; Neild & Balfanz, 2006). For high 
school students, attendance, course performance, credit attainment, and, in some cases, state assessment 
scores, grade retention, and behavior have frequently been identified as early indicators (Allensworth & 
Easton, 2005, 2007; Neild & Balfanz, 2006; Roderick, 1993; Silver, Saunders, & Zarate, 2008). 


While the research literature shows a high degree of consensus on the factors that place students at risk of 
failure to graduate on time, there is no guarantee that a given indicator will predict failure to graduate on time 
with the same accuracy if it is applied to students in different school contexts. Most previous studies have 
focused primarily on large urban centers, and even there, indicators’ value as predictors and cutpoints for indi- 
cators differ across districts (Hartman, Wilkins, Gregory, Gould, & D’Souza, 2011; Norbury et al., 2012). These 
findings suggest that the accuracy of indicators for predicting graduation outcomes may vary by context. 


What the study examined 


The three Ohio districts in this study were already implementing early warning systems to identify, provide 
services to, and track the progress of students in grades 8 and 9 who are at risk of failing to graduate from 
high school on time. The systems included the same indicators for all districts and for both grade 8 and 
grade 9. But as the literature suggests, indicators may not work equally well for different districts or for grade 
8 and 9 students within the same district. 


The purpose of this study was to develop a set of locally tailored early warning indicators for students at 
different grade levels in each school district and to examine the accuracy of the indicators for predicting 
failure to graduate on time (see box 1 for definitions of key terms used in the report). The study addressed 
four research questions: 
1. For each indicator, what is the optimal cutpoint for accurately classifying students as on track or off 
track to graduate? How do these cutpoints vary across districts and grade levels? 
2. Which indicators consistently predict failure to graduate on time? 
3. Which indicators are the most accurate in predicting failure to graduate on time? How does the 
accuracy of indicators vary across districts and grade levels? 
4. Is accuracy enhanced when students are identified as being at risk for not graduating on time using 
multiple indicators? 


The study team followed a five-step process (figure 1) and used student data—attendance records, tran- 
scripts, and discipline records, as well as graduation outcomes four years after entering grade 9—for two 
cohorts of grade 8 students and two cohorts of grade 9 students in three Ohio school districts (referred 
to as Districts A, B, and C) to identify district-specific early warning indicators of failure to graduate on 


Box 1. Key terms 


Accuracy. A statistic that captures how well an indicator correctly identifies students who graduate on time and 
students who do not (See appendix B of the full report for a description of how this statistic is calculated). Accu- 
racy statistics range from O to 1, with statistics greater than .50 indicating that using the indicator to identify 
students at risk of not graduating on time is better than randomly guessing. 


Combination indicator. One of three meta-indicators that flag students as off track to graduate on time if they 
are identified by at least one, at least two, or at least three consistent indicators. 


Consistent indicators. Indicators from earlier grade levels (grades 8 and 9 in this study) that show statistically 
significant relationships with students’ failure to graduate within four years of entering high school for the full 
sample and in the majority of iterations of regression models, each of which uses data from random samples of 
students (See appendix B of the full report for a description of this approach). 


Failure to graduate on time. The outcome measure used in this study. Students in each grade 8 and 9 cohort 
were Classified as not graduating on time if they did not receive a high school diploma within four years of begin- 
ning high school. Students who do not graduate within four years are classified as nongraduates even if they 
may later graduate. 


Optimal cutpoint. A specific value on the original scale of a grade 8 or 9 indicator that separates students who 
are at risk of not graduating on time from those who are not at risk. For example, if 90 percent is the optimal 
cutpoint for the grade 9 attendance indicator, students with an attendance rate at or below 90 percent could be 
classified as at risk of failure to graduate on time. For each indicator, the optimal cutpoint is the value with the 
highest accuracy. 


Figure 1. A systematic process for identifying valid early warning indicators of failure to graduate 
on time 


Step 1. Review the research. Review the research on early warning systems to identify Research-based 
candidate indicators that have been shown to predict failure to graduate on time. approach 
+ 
Step 2. Collect data on candidate indicators. Compile a longitudinal dataset with Available 


candidate indicators available in the district and graduation outcomes for two or 


more cohorts of grade 8 or 9 students. student data 


+ 
Step 3. Determine optimal cutpoints. For each candidate indicator, determine Optimal 
the threshold value that most accurately classifies students as on track or off hes oints 
track to graduate from high school on time (research question 1). P 
+ 
Step 4. Screen out weak and unreliable indicators. Eliminate indicators that do Consistent 
not consistently predict failure to graduate on time above and beyond the other redictors 
candidate indicators and student background characteristics (research question 2). P 
Step 5. Compare the accuracy of the consistent indicators. Calculate accuracy Valid set of 
statistics for the indicators that pass step 4, individually and in combinations of early warning 
two and three. Identify the indicators with the highest overall accuracy a cators 


(research questions 3 and 4). 


Note: Details about the methodology used to validate early warning indicators can be found in appendix B of Stuit et al. (2016). 


Source: Authors’ compilation. 


time. The three districts varied in size, demographic composition, and locale. Two of the districts serve 
large cities with populations greater than 250,000, while the third district serves a town near an urban 
area. One of the urban districts has more than 40,000 students, while the other districts each have 5,000 
to 10,000 students. The percentage of students qualifying for the federal school lunch program (a proxy for 
low income) also varied, ranging from about 40 percent to more than 90 percent. The four-year graduation 
rates for the three districts ranged from 56 percent to 91 percent. The analyses were restricted to first-time 
freshmen in each district in 2006/07 or 2007/08 and excluded students who entered the district after grade 
9. Students in the 2006/07 cohort graduated in 2010, and students in the 2007/08 cohort graduated in 2011. 


What the study found 


This section presents the findings for each of the study’s research questions. 
Optimal cutpoints for five of the eight indicators varied by district and grade level 


Research question 1 asked about the best cutpoints for the candidate indicators. The optimal cutpoints for 
five of the eight candidate indicators varied by district and grade level within districts (table 1). The two 
indicators with the most consistent cutpoints across districts and grade levels are the number of suspen- 
sions (one or more) and the number of failing grades in core courses (one or more). The optimal cutpoint 
for number of credits earned over the school year was also the same (fewer than seven) for the two districts 
that provided data on this indicator. 


The cutpoints for the other candidate indicators varied across districts and grade levels. For attendance rate 
the optimal cutpoints ranged from below 90 percent for grade 9 students in District A (which equates to 
missing 18 days in the state-required 182 schooldays) to below 95 percent for students in grades 8 and 9 in 
Districts B and C (9 of 182 days missed). 


For the number of failing grades (in core courses and overall), the cutpoint for 9 of the 10 district-grade 
combinations for which data were available was one or more failing grade in any course. The exception was 
for number of failing grades overall among grade 8 students in District B, for which the optimal cutpoint 


Table 1. On five of the eight data elements examined, the optimal cutpoint for designating 
students as off track to graduate differed for at least one district, 2006/07 and 2007/08 data for 
2010 and 2011 graduation outcomes 


District A District B District C 
Indicator Grade 8 Grade 9 Grade 8 Grade 9 Grade 8 Grade 9 
End-of-year attendance rate (percent) < 92 < 90 <95 <95 < 95 <95 
Grade point average — <19 < 2.3 < 2.1 <3.1 <2.2 
Number of failing grades overall >1 >1 >2 oa. 21 >1 
Number of failing grades in core courses >1 >1 >1 >1 _— _— 
Number of suspensions >1 >1 >1 >1 _— >1 
Math score on the Ohio Achievement Assessment < 392 na <406 na <413 na 
Reading score on the Ohio Achievement Assessment < 399 na <413 na <420 na 
Number of credits earned na <7 na <7 na oa 
na is not applicable for students at the indicated grade level; — is district did not provide student data or student characteristic or 


data were incomplete or unusable in the analysis. 


Source: Authors’ analysis of data on student attendance, achievement, coursework, and discipline provided by the three Ohio 
school districts in the study. 


was two or more failing grades. The optimal cutpoints for grade point average also varied, ranging from 
below 1.9 (grade 9 in District A) to below 3.1 (grade 8 in District C). 


The optimal cutpoints for grade 8 reading and math scores on the Ohio Achievement Assessment also varied 
across districts. In District A the cutpoint was 392 for math and 399 for reading. In District B the cutpoint was 
406 for math and 413 for reading. And in District C the cutpoint was 413 for math and 420 for reading. 


While the study identified at least two consistent predictors of failure to graduate on time for each school district and 
grade level, the specific predictors found to be consistent across districts and grade levels varied 


The study team addressed research question 2 by conducting separate sets of iterative logistic regression 
models using grade-specific data from each district.' The analyses identified between two and six consistent 
predictors of failure to graduate on time per district and grade level (table 2). 


Of the eight candidate indicators examined, only end-of-year attendance rate was available in every district 
and grade level and consistently predicted failure to graduate on time. Specifically, when attendance rate 
was below the district and grade-specific cutpoint, students were more likely to not graduate on time. 
Reading score on the Ohio Achievement Assessment (a candidate indicator found only in grade 8 data) 
was a consistent predictor in all three districts. Students with reading scores below the cutpoint were more 
likely to not graduate on time. Number of credits earned, an indicator found only in grade 9 data and only 
in two districts, also consistently predicted failure to graduate on time, so students whose credit earnings 
fell below the cutpoint at the end of grade 9 were less likely to graduate on time. 


Other indicators were consistent predictors of failure to graduate on time for some school districts and grade levels 


Other indicators’ ability to consistently predict failure to graduate on time varied by grade level and district: 

e For four of five grade—district combinations for which data were available, students whose grade 
point average was below the respective cutpoints were consistently less likely to graduate on time. 

e For three of six grade—district combinations, students whose number of failing grades overall and 
number of suspensions were above their respective cutpoints were less likely to graduate on time. 

e For two of four grade—district combinations, students whose number of failing grades in core 
courses was above the cutpoint were less likely to graduate on time. 

e For one of the three districts, students whose math score on the Ohio Achievement Assessment 
was below the cutpoint were less likely to graduate on time. 


The most accurate indicators of failure to graduate on time also varied by school district and grade level 


Research question 3 asked about the accuracy of the consistent predictors.? The accuracy with which the 
consistent predictors identify students at risk of not graduating on time also varies by district and grade level. 
Attendance rate, grade point average, number of credit earned, reading score on the Ohio Achievement 
Assessment, and combination indicators had the highest accuracy rates (table 3; see tables 2—4 in the full 
report for detailed findings, including accuracy statistics for different districts and grade levels). 


Single indicators generally outperformed combined indicators 


Research question 4 asked whether accuracy was enhanced if the focus was on combination indicators 
instead of on a specific indicator. Accuracy was greater for combination indicators in only one of the six 
grade—district combinations (grade 9 in school district C). In all other cases, one can maximize the accura- 
cy of prediction by looking at a single specific indicator (see bottom of table 3). 


Table 2. The indicators that consistently predicted failure to graduate on time varied by school 
district and grade level, 2006/07 and 2007/08 data for 2010 and 2011 graduation outcomes 


District A District B District C 

Indicator Grade 8 Grade 9 Grade 8 Grade 9 Grade 8 Grade 9 
End-of-year attendance rate * * * - * * 
Grade point average _— * + * * 
Number of failing grades overall * ba * 

Number of failing grades in core courses * * _ _ 
Number of suspensions * * * —_— 

Math score on the Ohio Achievement Assessment i na na na 
Reading score on the Ohio Achievement Assessment * na * na * na 
Number of credits earned na = na = na _— 


* Statistically significant at p < .05 in 50 or more of the 100 simulated cohorts of students. 


na is not applicable for students at the indicated grade level; — is district did not provide student data or student characteristic or 
data were incomplete or unusable in the analysis. 


Note: Indicators were classified consistent if they were statistically significant when other candidate indicators were included in 

models and when they predicted failure to graduate on time in 50 or more simulated cohorts of students in that grade level. Sim- 
ulated cohorts represented a random selection of students from the cohorts studied for that district. Blank cells indicate that the 
indicator did not consistently predict failure to graduate on time. 


Source: Authors’ analysis of data on student attendance, achievement, coursework, and discipline provided by the three Ohio 
school districts. 


Table 3. The overall accuracy of indicators also varied by district and grade level, 2006/07 and 
2007/08 data for 2010 and 2011 graduation outcomes 


District A District B District C 
Indicator Grade 8 Grade 9 Grade 8 Grade 9 Grade 8 Grade 9 
Consistent indicators 
End-of-year attendance rate .67 .69 64 .67 69 1 
Grade point average —_— 13 10 10 15 
Number of failing grades overall 65 .68 10 
Number of failing grades in core courses .62 10 _— — 
Number of suspensions 57 .60 .68 —_ 
Math score on the Ohio Achievement Assessment 62 na na na 
Reading score on the Ohio Achievement Assessment 64 na .66 na 12 na 
Number of credits earned na .68 na 14 na — 


Combination indicators 


Off track on at least one consistent indicator .62 .64 .63 10 .66 14 
Off track on at least two consistent indicators .66 .66 .67 12 .69 18 
Off track on at least three consistent indicators .66 By Gl 10 12 .67 

na is not applicable for students at the indicated grade level; — is district did not provide student data or data were incomplete or 


unusable in the analysis. 


Note: Proportions in the cells represent the accuracy with which indicators predicted failure to graduate on time. The greater the 
proportion, the greater the percentage of students who were correctly classified as off track to graduate on time. Blank cells indi- 
cate that the indicator did not consistently predict failure to graduate on time. 


Source: Authors’ analysis of data on student attendance, achievement, coursework, and discipline provided by three Ohio school 
districts. 


Implications of the study findings 


This analysis was intended to help three Ohio school districts make well-informed choices about which 
indicators to use in their early warning systems and to provide the districts with clear expectations about 
the accuracy of their early warning indicators when applied to current and future cohorts of students. The 
process of identifying grade 8 and 9 indicators that consistently predict failure to graduate on time can 
serve as a model for other states and district eduation agencies when they are designing their own early 
warning systems. 


Applying indicators that are based on analysis from a different school district may lead to the wrong students being 
identified as off track for graduating 


The indicator cutpoints that most accurately identified students as off track to graduate on time varied 
by school district. When the optimal cutpoints were used, some indicators were found to be consistent 
and accurate predictors of failure to graduate on time in one district but not in the others. These findings 
underscore why it is valuable for districts to analyze their own data and identify the best indicators for their 
early warning system or at least to verify that the indicators used are valid for their student population. 
For example, if District A’s 90 percent optimal cutpoint for attendance in grade 9 were applied to grade 9 
students in District B instead of District B’s optimal cutpoint of 95 percent, it would reduce the number of 
students who are correctly flagged as off track to graduate on time from 67 percent to 25 percent. 


Availability and accuracy of indicators also vary by grade level 


Because of the timing of data collection at different points in students’ grade-level progressions, the candi- 
date indicators that are the most consistent and accurate predictors will inevitably vary by grade level. The 
most obvious example involves math and reading scores on the Ohio Achievement Assessment in grade 
8. This assessment was not administered to students in grade 9 and thus was not included in prediction 
models using candidate indicators in the grade 9 data. Likewise, the accumulation of credits toward high 
school graduation does not begin until grade 9, making it a grade-specific indicator as well. 


The data suggest that accuracy is higher for indicators found in grade 9 student records than for those found 
in grade 8 student records. This is evident in the higher accuracy for the combination indicator off track on 
at least one consistent indicator. The difference in accuracy between indicators for grade 8 and 9 students 
was 0.02 for District A, 0.07 for District B, and 0.08 for District C. The higher accuracy rates reflect the 
increased stakes for grade 9 students, since grade 9 students’ academic performance counts toward gradua- 
tion, whereas grade 8 students’ academic performance is less likely tied to graduation requirements. Grade 
8 students also have an additional year to improve on past poor performance. 


This grade-level differential in accuracy does not imply that districts’ early warning systems should rely solely 
on indicators taken from student's grade 9 records. Instead, the accuracy statistic reflects only the percentage 
of students who were correctly classified as off track to graduate. Indeed, districts using grade 8 data to identify 
at-risk students will have more time to intervene with those students and get them back on track to graduate. 


When selecting indicators, school districts need to consider the risk of over- or underprediction 


In the three district- and grade-specific datasets for which one or more suspensions was a consistent pre- 
dictor, the indicator had the lowest rate of misclassification of students as off track to graduate on time 
(9-15 percent; see tables 2—4 in the main report). This may make the indicator attractive for districts with 
limited resources that want to target a subgroup of students who are at high risk of failure to graduate on 


time. However, the drawback of suspension indicators is that they greatly underpredict the number of stu- 
dents who are off track because they miss those who are staying out of trouble but still failing academically. 
Across all three districts, fewer than 50 percent of nongraduates were correctly designated as off track by 
these indicators. 


Using a combination indicator that flags all students who are off track on any one or more indicators cap- 
tures the most students who are truly off track to graduate, but it comes at the cost of incorrectly flagging a 
high percentage of students who are not at risk of failure to graduate on time. This catch-all approach may 
make sense for districts that desire to intervene with all students who are truly at risk of failure to graduate 
on time, even if it means that a higher percentage of the students who receive dropout prevention services 
may not need them. Districts that want to focus on students who are most at risk may want to use a combi- 
nation of at least two or three indicators. 


Because districts operate within budget and human resource constraints, selecting indicators that over- 
predict the number of students who will not graduate on time will result in supplemental services being 
provided to students who do not need them. However, if an early warning system’s goal is to catch as many 
at-risk students as possible, districts might want to avoid underprediction and identify as many potentially 
at-risk students as possible, even if it means including some students who are not actually at risk. Choosing 
an early warning system that balances these values ultimately depends on the local context and the sys- 
tem’s intent. 


Limitations of the study 


This study has several limitations. 


First, the analysis focused on indicators that were consistently identified as predictive of failure to grad- 
uate on time in the research literature on early warning systems and were based on data elements that a 
typical school district routinely collects and stores. Other data elements on grade 8 and 9 students—such 
as whether students are overage for grade, are eligible for special education services, are English learner stu- 
dents, or participate in extracurricular activities—also may accurately predict failure to graduate on time. 
School districts can follow the analytic process outlined in this report to determine whether these other 
candidate indicators consistently predict failure to graduate on time. 


Second, this study used only the four-year graduation rate as the outcome because it was of most interest 
to the members of the Midwest Dropout Prevention Research Alliance and is the accountability measure 
for high schools tracked by the state of Ohio. Some studies suggest that indicators predicting four-year 
graduation rates may be equally good at predicting five-year graduation rates (Allensworth & Easton, 2005). 
Additional data and analysis would be needed to verify whether the indicators of four-year graduation rates 
also predict five-year graduation rates for these Ohio districts. 


Third, the analysis was restricted to students who were first-time freshmen within the districts in 2006/07 
or 2007/08 and excluded students who entered the district after grade 9. Thus, this study does not present 
a full portrait of the graduating classes of 2010 and 2011. Furthermore, findings based on retrospective 
cohorts might not apply to current and future cohorts. 


Fourth, results from the three participating districts (which were not randomly selected) may not generalize 
to other districts in Ohio or other states. Although the methods presented here for validating early warning 
indicators are applicable to other districts, the specific findings on the candidate indicators may not apply 
to other districts. 


Finally, this study has two conceptual caveats. First, the study focuses on indicators of failure to graduate 
on time rather than on indicators of whether students identified through early warning systems respond 
positively to the interventions designed to get them back on track. Additional research is needed to iden- 
tify indicators of whether students respond to graduation-focused interventions. Second, the relationships 
between indicators and students’ graduation outcomes are only correlational. The findings say nothing 
about a causal connection between indicators and later outcomes. Early warning indicators only distin- 
guish students who have a low probability of on-time graduation. 


Notes 


Whether indicators were consistent predictors was determined through a two-step process, each 
involving logistic regression. First, for each district and grade level, indicators had to show a statistically 
significant relationship with failure to graduate on time, even when all other candidate indicators were 
included in the statistical model (statistical significance was set at p < .05). Second, the indicator had 
to show statistical significance for at least 50 of the 100 randomly drawn subsamples of students. Addi- 
tional details on analytic methods are available in the full report. 

An indicator’s accuracy was calculated by first calculating the correct off-track prediction rate (the 
proportion of students who were identified as at risk and actually failed to graduate) and the false alarm 
rate (the proportion of students who were identified as at risk but actually graduated on time). A high 
accuracy rate for an indicator (close to 1.00) suggests a combination of a higher correct off-track predic- 
tion rate and a lower false alarm rate. See the full report for more details. 
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